Add v2.1 BGC data #56

forsyth2 · 2025-03-12T20:40:52Z

Resolves #55. Add v2.1 BGC data. The link, once merged, will be https://docs.e3sm.org/e3sm_data_docs/_build/html/v2.1/BGC.

HPSS path for data: /home/projects/e3sm/www/BGC/E3SMv2_1
Simulation metadata: https://acme-climate.atlassian.net/wiki/spaces/ED/pages/2766340117/V2+Simulation+Planning

forsyth2 · 2025-03-12T20:44:10Z

To do:

Get metadata to fill out new land rows in utils/simulations_v2_1.csv
Add the original run scripts to run_scripts/v2.1/original/
Use utils/generate_html.bash to generate a table for https://docs.e3sm.org/e3sm_data_docs/_build/html/v2.1/BGC/simulation_data/simulation_table.html , similar to https://docs.e3sm.org/e3sm_data_docs/_build/html/v2.1/WaterCycle/simulation_data/simulation_table.html
Code review

forsyth2

@BunnyVon (and cc'ing @chengzhuzhang as a second reviewer for this PR) I have an initial web page -- preview at https://portal.nersc.gov/cfs/e3sm/forsyth/data_docs_56/html/v2.1/BGC/simulation_data/simulation_table.html:

Remaining questions:

Do any of the 3 simulations have/need ESGF links?
(see comments for remainder)

run_scripts/v2.1/original/run.CBGCv2.r05.chrysalis.I20TRGSWCNP.sh

run_scripts/v2.1/original/run.v2.LR.BGC-LNDATM.CONTRL.sh

run_scripts/v2.1/original/run.v2.LR.BGC-LNDATM.SSP370.sh

utils/simulations_v2_1.csv

forsyth2 · 2025-03-12T22:34:11Z

utils/simulations_v2_1.csv

-v2.1, WaterCycle, LR, Historical, v2_1.LR.historical_0251,      chrysalis,                                 , historical,   4, cmip_only, cels.anl,
-v2.1, WaterCycle, LR, Historical, v2_1.LR.historical_0301,      chrysalis,                                 , historical,   5, cmip_only, cels.anl,
+model_version, group, resolution, category, simulation_name, machine, checksum, experiment, ensemble_num, cmip_only, node,
+v2.1, BGC,        LR, SPINUP,     CBGCv2.r05.chrysalis.I20TRGSWCNP, chrysalis,                                 ,             ,  ,          ,         ,


So, we still want this data grouped with v2.1 even though the cases are prefixed with v2 (not v2.1), and this simulation in particular seems to use a completely different naming convention?

BunnyVon · 2025-03-12T22:47:33Z

To do:

Get metadata to fill out new land rows in utils/simulations_v2_1.csv

Add the original run scripts to run_scripts/v2.1/original/

Use utils/generate_html.bash to generate a table for https://docs.e3sm.org/e3sm_data_docs/_build/html/v2.1/BGC/simulation_data/simulation_table.html , similar to https://docs.e3sm.org/e3sm_data_docs/_build/html/v2.1/WaterCycle/simulation_data/simulation_table.html

Code review

All runscripts are placed in chrysalis:/lcrc/globalscratch/ac.sfeng1/archive/runscript

forsyth2

@chengzhuzhang -- @BunnyVon would like to have this merged by end of day Friday (3/14). Please review if possible, and if you don't have time, I can go ahead and merge this (after some changes detailed below); we can always update the web pages later.

@BunnyVon added more data to one of the archives, so I will rerun generate_html.bash before merging so the data size column gets updated.

Answers to my remaining questions:

1 & 5 -- @BunnyVon said ESGF links are not necessary.
6 -- @BunnyVon confirmed this is correct; I will add a note in docs/source/v2.1/BGC/index.rst about how BGC is .1 behind in naming its simulations

chengzhuzhang · 2025-03-14T15:57:00Z

-- preview at https://portal.nersc.gov/cfs/e3sm/forsyth/data_docs_56/html/v2.1/BGC/simulation_data/simulation_table.html:

I think this looks great by reviewing the preview page.

forsyth2 · 2025-03-14T16:05:36Z

Thanks @chengzhuzhang!

forsyth2 · 2025-03-14T16:26:25Z

Ok, I made my updates and will now merge this.

The only remaining issue is that the "NERSC HPSS web interface" link works for Water Cycle, but not for BGC: "NERSC HPSS web interface". I believe the fix for that is to make the simulation directories world-readable.

forsyth2 · 2025-03-14T16:40:09Z

https://portal.nersc.gov/archive/home/projects/e3sm/www/BGC/E3SMv2_1 works now, looks like the directory permissions change worked.

forsyth2 · 2025-04-16T20:54:44Z

@chengzhuzhang I just noticed that @BunnyVon's link is https://portal.nersc.gov/cfs/e3sm/forsyth/data_docs_56/html/v2.1/BGC/index.html, which is in my individual directory. Furthermore, the page doesn't seem to exist where it should in the real docs: https://docs.e3sm.org/e3sm_data_docs/_build/html/v2.1/index.html

My individual directory:

docs.e3sm.org:

I thought merging the PR here triggered a documentation build, so I'm confused as to why docs.e3sm.org isn't updated. This PR more or less matches #50, which did manage to update docs.e3sm.org...

chengzhuzhang · 2025-04-18T19:37:47Z

the gh-page is not built with this change:https://github.com/E3SM-Project/e3sm_data_docs/tree/gh-pages
Maybe try re-build?

forsyth2 · 2025-05-12T21:46:05Z

Yes, it looks like the build did fail: https://github.com/E3SM-Project/e3sm_data_docs/actions/runs/13861015009/job/38789265167#step:1:37:

Error: Missing download info for actions/cache@v2

But I'm not seeing a good way to re-run it. Running the workflow on main just skips it and there's no rerun button on the failing build. I also found https://github.blog/changelog/2024-12-05-notice-of-upcoming-releases-and-breaking-changes-for-github-actions/ , which states:

Starting February 1st, 2025, Actions’ cache storage will move to a new architecture, as a result we are closing down v1-v2 of actions/cache. In conjunction, all previous versions of the @actions/cache package (prior to 4.0.0) in actions/toolkit will be closing down. The action and cache package will be fully retired on March 1st.
If users run workflows that call the retired versions after March 1st, 2025, the workflows will fail. Announcements have been posted in the actions/cache and actions/toolkit repositories with additional information on the migration. Note that this does not affect GitHub Enterprise Server customers, you can continue to use all versions without failure.

@tomvothecoder Have you run into this? I'll keep digging into it, but figured I'd ask as well.

tomvothecoder · 2025-05-12T22:01:08Z

Yes, it looks like the build did fail: E3SM-Project/e3sm_data_docs/actions/runs/13861015009/job/38789265167#step:1:37:
Error: Missing download info for actions/cache@v2
But I'm not seeing a good way to re-run it. Running the workflow on main just skips it and there's no rerun button on the failing build. I also found github.blog/changelog/2024-12-05-notice-of-upcoming-releases-and-breaking-changes-for-github-actions , which states:

Starting February 1st, 2025, Actions’ cache storage will move to a new architecture, as a result we are closing down v1-v2 of actions/cache. In conjunction, all previous versions of the @actions/cache package (prior to 4.0.0) in actions/toolkit will be closing down. The action and cache package will be fully retired on March 1st.
If users run workflows that call the retired versions after March 1st, 2025, the workflows will fail. Announcements have been posted in the actions/cache and actions/toolkit repositories with additional information on the migration. Note that this does not affect GitHub Enterprise Server customers, you can continue to use all versions without failure.

@tomvothecoder Have you run into this? I'll keep digging into it, but figured I'd ask as well.

No I have not, but probably because I updated build workflows to actions/cache@v3 at some point. This repo still uses actions/cache@v2, which looks like is deprecated.

Try opening a new PR to update the build work flow to use actions/cache@v4 and see what happens.

EDIT: Nevermind I just saw you tried that in #58.

forsyth2 · 2025-05-12T22:18:39Z

Thanks @tomvothecoder, it looks like #58 got us working -- https://docs.e3sm.org/e3sm_data_docs/_build/html/v2.1/BGC/index.html

Add v2.1 BGC data

94d30e2

forsyth2 self-assigned this Mar 12, 2025

Generated table

b90af59

forsyth2 commented Mar 12, 2025

View reviewed changes

Add original run scripts

4f488a0

forsyth2 force-pushed the issue-55-bgc-data branch from 9a7c913 to 4f488a0 Compare March 12, 2025 23:03

forsyth2 commented Mar 14, 2025

View reviewed changes

Address comments

30a568b

forsyth2 marked this pull request as ready for review March 14, 2025 16:27

forsyth2 merged commit 2c942ae into main Mar 14, 2025
1 check passed

forsyth2 deleted the issue-55-bgc-data branch March 14, 2025 16:27

forsyth2 mentioned this pull request May 12, 2025

Fix build issue with cache #58

Merged

Add v2.1 BGC data #56

Add v2.1 BGC data #56

Uh oh!

Conversation

forsyth2 commented Mar 12, 2025

Uh oh!

forsyth2 commented Mar 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

forsyth2 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

forsyth2 Mar 12, 2025

Choose a reason for hiding this comment

Uh oh!

BunnyVon commented Mar 12, 2025

Uh oh!

forsyth2 left a comment

Choose a reason for hiding this comment

Uh oh!

chengzhuzhang commented Mar 14, 2025

Uh oh!

forsyth2 commented Mar 14, 2025

Uh oh!

forsyth2 commented Mar 14, 2025

Uh oh!

Uh oh!

forsyth2 commented Mar 14, 2025

Uh oh!

forsyth2 commented Apr 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

chengzhuzhang commented Apr 18, 2025

Uh oh!

forsyth2 commented May 12, 2025

Uh oh!

tomvothecoder commented May 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

forsyth2 commented May 12, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

forsyth2 commented Mar 12, 2025 •

edited

Loading

forsyth2 commented Apr 16, 2025 •

edited

Loading

tomvothecoder commented May 12, 2025 •

edited

Loading